docs(distributed): review frozen multimodal FSDP guidance - #3272
Merged
Conversation
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
Contributor
jgerh
reviewed
Jul 29, 2026
jgerh
left a comment
jgerh
left a comment
Contributor
There was a problem hiding this comment.
Completed tech pubs review. No changes needed to docs/guides/configuration.mdx - looks great. Provided minor edits to docs/breaking-changes.mdx.
|
|
||
| {/* docs-review-start: frozen-multimodal-fsdp-policy */} | ||
|
|
||
| Fully frozen vision/audio towers and multimodal projectors now default to the `root` FSDP2 policy. Previously, dense |
Contributor
There was a problem hiding this comment.
Suggested change
| Fully frozen vision/audio towers and multimodal projectors now default to the `root` FSDP2 policy. Previously, dense | |
| Fully frozen vision and audio towers and multimodal projectors now default to the `root` FSDP2 policy. Previously, dense |
|
|
||
| ### FSDP2 Default `reduce_dtype` Is Now `float32` | ||
|
|
||
| The default [`MixedPrecisionPolicy`](https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html) built by `FSDP2Config` now uses `reduce_dtype=torch.float32` instead of `torch.bfloat16`. Forward/backward compute still uses `param_dtype=torch.bfloat16`, but gradient reduction now accumulates in fp32 to reduce communication-rounding error at larger data-parallel world sizes. |
Contributor
There was a problem hiding this comment.
Suggested change
| The default [`MixedPrecisionPolicy`](https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html) built by `FSDP2Config` now uses `reduce_dtype=torch.float32` instead of `torch.bfloat16`. Forward/backward compute still uses `param_dtype=torch.bfloat16`, but gradient reduction now accumulates in fp32 to reduce communication-rounding error at larger data-parallel world sizes. | |
| The default [`MixedPrecisionPolicy`](https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html) built by `FSDP2Config` now uses `reduce_dtype=torch.float32` instead of `torch.bfloat16`. Forward/backward compute still uses `param_dtype=torch.bfloat16`, but gradient reduction now accumulates in FP32 to reduce communication-rounding error at larger data-parallel world sizes. |
Contributor
There was a problem hiding this comment.
Suggested change
| | Kubernetes (through SkyPilot) | `skypilot:` with `cloud: kubernetes` | |
Comment on lines
111
to
112
Contributor
There was a problem hiding this comment.
Suggested change
| The `slurm:` YAML section and all related fields have been removed. Slurm | |
| jobs are now submitted with `sbatch` directly, using a self-contained `sbatch` |
Comment on lines
123
to
124
Contributor
There was a problem hiding this comment.
Suggested change
| All cluster-specific configuration lives in the `sbatch` script where you can | |
| edit it directly. |
Contributor
There was a problem hiding this comment.
Suggested change
| The FFmpeg-bearing media dependencies (OpenCV, decord, the Qwen vision utilities, and |
Comment on lines
156
to
157
Contributor
There was a problem hiding this comment.
Suggested change
| - `[vlm]` alone no longer trains Qwen2.5-VL, Qwen3-Omni, or Mistral VLMs. Add `vlm-media`. | |
| - `[all]` no longer includes the media extras. Add them with `uv pip install "nemo-automodel[media]"` in the activated environment. |
Comment on lines
163
to
164
Contributor
There was a problem hiding this comment.
Suggested change
| The CLI entry point lives at `nemo_automodel/cli/app.py` and is registered as | |
| the `automodel` or `am` console entry points. A thin convenience wrapper |
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
Contributor
Author
|
/ok to test 4560ee1 |
yuhezhang-ai
added
the
r0.6.0
akoumpa
approved these changes
Jul 30, 2026
akoumpa
pushed a commit
that referenced
this pull request
Jul 30, 2026
…` into `r0.6.0` (#3306) docs(distributed): review frozen multimodal FSDP guidance (#3272) * docs(distributed): mark frozen multimodal sections for review * docs(distributed): apply technical publications review --------- Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Yuhe Zhang <yuhez@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This docs-only follow-up completes the technical publications review of the customer-facing documentation introduced by #2763.
docs/breaking-changes.mdx.docs/guides/configuration.mdxunchanged after technical publications review found no edits were needed.The final PR diff contains only the accepted editorial updates to
docs/breaking-changes.mdx.Validation
make -C docs/fern docs-check— MDX validation passed for 452 files; the local Fern CLI was unavailablenpx -y fern-api@5.29.0 check— blocked by host WebAssembly memory allocation; PR Fern CI provides the complete checkgit diff --check origin/main...HEAD— passedRequested by @jgerh in #2763.